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Agenda 


e Introduction to Liquid Cooling 
e Case studies 
e Interactive Discussion 
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Liquid Cooling Solution præ 


Buildini ng 


Equipment center 
Cabinet 


a 


Typical liquid cooled equipment room, with external coolant distribution units (CDUs) 
For most locations these data centers may be operated without chillers in a water-side economizer mode 
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Benefits of Liquid Cooling 


e Higher compute densities 
* Higher efficiency 
e Heat removal 
e Transport energy 
e cooling plant 
e Increased economizer hours 
e Potential use of waste heat 
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Moving (Back) to Liquid Cooling 


As heat densities rise, liquid solutions become more attractive 
Volumetric heat capacity comparison (5,380 m?) 


Water 
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Why Liquid Cooling? 


= Higher compute densities 
= Liquids can provide cooling at higher temperatures 


* Improved cooling efficiency 
* |ncreased economizer hours 
* Potential use of waste heat 


= Reduced transport energy: 


Rate Heat Transfer Fluid Flow | Conduit | Theoretical 
Medium D É— D —=—=_ 


FOR Al 9217 d Feat ve EEN 
10 Tons | 12°F | ia 
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In-Rack Liquid Cooling 


Racks with integral coils and full containment: 


Fan ^ 
-Cool Air 
Hot Air J 
IT 
Equipment 
in Rack 
Server cabinet technology 
with outstanding benefits 
. up to 35kW cooling capacity 
- Cooling 
Coil 


Blade server = m 
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Rear-Door Heat Exchanger 


= Passive technology: relies 
on server fans for airflow 


= Active technology: 
supplements server fans 
with external fans in door 


= Can use chilled or higher 
temperature water for 
cooling 
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Liquid On-Board Coolin 


* Clustered Systems design 


" Conducting heat to a cold sa | A | Vr 
plate containing refrigerant aset en 
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Liquid On-Board Cooling 


= Server fans are removed 
= Heat risers connect to the top plate, which has a micro channel heat exchanger 


Compliant Pad 


on server lid 
Heat conductor Simple heat 


Gap Filler conductor Gap filler DRAM Server Lid 
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Liquid On-Board Cooling 
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Liquid Immersion Cooling 


Vapor Condenser 


Boiling 
Liquid 
Server A 
Facility Water 
Ambient as 
ja high as 55*C 
Tank 


Facility 


wall Pump 
— ACT 
IT Power Cooling Power — Pump + Fan 


No longer requires: 
e chillers * raised floors 


e cooling towers e computer room air conditioners 


e water use e earplugs! 
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Phase Change of Dielectnc 
Fluid Removes Heat Efficiently 


Computer in glass tank 


3 Server € 
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"Chill-Off 2" Evaluation of Liquid 


Cooling Solutions 


1.55 


COEEC(kW/kW) 
kA 
D 


Data Center Cooling Device Relative Performance 


ge A CRAC 


w/ DX comp. 
CRAH 


w/ Chilled water 
CRAHEstimate Rack 


l Eoi Cooler 
wen, 4— Rear Door 
Heat Exchanger 


(passive) 


server air inlet temp. ČF) 60 


1.05 
Direct Touch 
— .. —— 
Cooling 
0. 95 T T T T T T 
test ID# 1 2 3 4 5 6 7 
chilled watertemp. ČF) 45 45 50 55 60 60 60 


72 72 72 72 80 90 


Test ID Number - Test Parameters 
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"Free" Cooling w/ Water- 


Side Economizers 


= Cooling without Miu 
Compressors compressor 


cooling 


» Easier retrofit 


= Added reliability 
EE in case of 
a 


chiller failure 
É ) . Cooling tower and HX = Water-side Economizer 
* No contamination 
issues 4 à 
* Put in series with | L 
chiller 
= Uses tower or dry lis 
cooler å F^ ec? 
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Potential for Tower Coolinc 


On-Board Cooling 


Water-Side Economizer 


Sacramento 
Los Angeles 


Oakland 
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Sacramento 


Oakland 


Free Cooling Potential (Hours per Year) 
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F Chilled Water Design Temperature (Degrees F) d 
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Re-Use of Waste Heat 


= Heat from a data center can be used for: 
* Heating adjacent offices directly 
* Preheating make-up air (e.g., "run around coi 


|" 


for adjacent laboratories) 


= Use a heat pump to elevate temperature 
* Waste heat from LBNL ALS servers captured with rear door coolers feed a heat pump that provides 
hot water for reheat coils 
= Warm-water cooled computers are used to heat: 
* Greenhouses, swimming pools, and district heating systems (in Europe) 
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ASHRAE Design Reference Con 


2015 ASHRAE Equipment Environmental Specifications far Air Cooling 
Product Operations Product Power Off 


Dry-Bulb Maximum | Maximum Maximum Dry-Bulb Ň 
Humidity Range, Ň Relative 
Temperature Ň Dew Point Elevation | Temperature Range | Temperature uS 
å é » Non-Condensing D ft [15 min/1 hr] "F » Humidity 96 
Air side 


Conditions 15,8“F DP to 59°F DP 
Al to A4 | 68.4 to 80.6 pg Pasa 


Legacy equipment 10.4*F DP & 8% RH to 
ges ` KS echt) SM Ei 10,006 9/36 41to113 | 8t080 

10.4"F DP & 8*6 RH to 
50 to 95 69.8"F and 80% RH Ei 10,006 9/36 41 to 113 8 to 80 


10.4'F DP & 8% RH to 
b 4 
41 ta 113 75.2"F and 90% RH 10,006 9/36 1to 113 ato 80 
855 to 82.4*F DP and 
. 9, 4 
EX er 80% RH — iini ae 
8% to 82.4°F DP and i 
2,4 10, 41 11: 0 
| © Town | 80% RH s 9/36 diia 


Water side Liquid Cooling Typical Infrastructure Design 


New racks with rear —— 
door ht exchangers 


Facility Supply Water Temperature 


35.6°F to 62.6°F 


Conditions Classes Main Heat Rejection Equipment Supplemental Cooling Equipment 


New Racks 
(W2 Condition - with the På 
exception of delivering 84 deg 


F water on peak WB days) Cooling Tower 


Chiller 35.6°F to 89.6°F 


Water-side Economizer 1 " " 
- (With Drycooler or Cooling Tower) - undis 
Building Heating System Cooling Tower >113°F 
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Liquid Cooled Rack Specification 


= While liquid cooling potential is understood, uptake is slow 
= Most solutions are unique and proprietary 
= Needed: 


e Multi-source solution 
e Reusable rack infrastructure 


= Users can drive faster technology development and 
adoption | 
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GoalforLiquid Cooled RackSpecifications 


= Goal: 


* A liquid cooled rack specification that could accommodate multiple 
vendors and provide a reusable infrastructure for multiple refresh cycles 
with a variety of liquid cooled servers/suppliers. 


= The working group has focused on the following 
specifications: 
° lt) wetted material list (all components must be compatible with this 
ist 
* Water based transfer fluid quality and treatment 
* Auniversal (multi-vendor) quick connect 
* General operating specifications such as pressure and temperature 
ranges 
= https://datacenters.Ibl.gov/technologies/liquid-cooling 
ee. or MG NNN 
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Intemational Open Data Center Specifications 


Target Organizations: 
e The Open Compute Project (OCP) — US 


Collaborators: 
* Facebook 
* Google 

Intel 

* Microsoft 

* Baidu 

* Alibaba 

* Tencent 

e YOU 
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Resources 


* Center of Expertise for Energy Efficiency in Data Centers 


e  datacenters.lbl.gov/ 


e https://datacenters.|bl.gov/technologies/liquid-coolin 
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Liquid Cooling at 
NERSC: 


Good Design & Regional. 


Giimate make for 


Wc M 
What is NERSC ? 


= NERSC is the National Energy Research Scientific 
Computing Center — 


= Founded in 1974 
= Focused on open science 


= Located at Lawrence Berkeley National Laboratory 


= Operated for the U.S. on Of Energy [ROE 
Office Bi Science 
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NERSC ProvidesHPC and Data Resourc es 
for DOE Office of Science Research 


me, U.S. DEPARTMENT OF Office of ` Largest funder of physical 


) EN ERGY |: Science science research in U.S. 


Materials, Chemistry, 
Geophysics 


Particle Physics, Nuclear Physics Fusion Energy, 
Astrophysics 777000 Users world wide Plasma Physics 
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Moved from Oakla nd Site in 2015 


= 11 MW power (6+5MW) 
= 19,000 ft“, 250 Ibs/ft? 
= Chilled Air + Water 


: PUE ^ 1.3 


= Extensive environmental and 
energy-usage monitoring 


ZA] . ESnet © 10Gb/s & 100Gb/s 
= Control room staffed 24 X 365 
= 6.5 miles from LBNL main campus 
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Shyh Wang Hall (B59) completed J un 2015 


* Four story 150,000 GSF 
* Two 20ksf office floors, 300 offices 
e 20k -> 28ksf HPC floor 
e 12.5MW expandable to 20 MW 
* Current demand is "7MW 


= Energy Efficient 


e Year-round compressor free air and 
water cooling 


e LEED Gold 
e PUE<1.1 


= Extensively Instrumented 
e Substations, panels, PDUs, UPS 
* Cray Systems SEDC 
* One-wire Temp & RH sensors 

* BMS through ALC/BACNET 


Indoor Particle counters 
Weather station 
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Water Cooling 
* Cooling towers 
* (3+1) x 3.375MW cooling towers 
e Expansion room to Qty. 7 total 
* Cooling towers operate every day 


= 74°F water to racks and AHUs 
* 65°F water for 92% of the year 
* 65-70*F water for 5-696 of the year 
* 70-74°F water for 2-396 of the year 


= Water Consumption 
* At 7.5MW 18M gallons/yr 
* At 17MW, estimated to be 40M gallons/yr 
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all yearround 


75.0 4 


70.0 + 


Minimum ECW Supply Temperatrue (° F) 


Supplied water 


temp 

(hours per vear) 

One Cell 8258 
Two Cell 8636 


75 


100 125 150 175 
Day of Year ns at Jan 1 


70-71F 71-72F 72-73F 73-74F 


261 


63 32 19 7 
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Climate providesthe ability to generate cool airand water 
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Liquid Cooling Performance Baseline 


300 


250 


Power (kW) 
3 


g 


100 


49 51 53 55 57 59 61 63 
TW Supply Temperature (°F) 


kW 


+ Cooling Loop Pump kW + Tower Water Pump kW * Cooling Tower Fan kW * Total kW 
ENGINEERING 
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Liquid Cooling Performance Balanced 


350 
300 


250 


S 


Power (kW) 
I 
un 
e 


100 


50 


0 T I | — - | — T | T I ] 
49 51 53 55 57 59 61 63 


kW TW Supply Temperature 


ENGINEERING + Cooling Loop Pump kW + Tower Water Pump kW Cooling Tower Fan kW + Total kW 
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Cray XC Senes- Liquid Cooled 


Edison and Cori are XC Series Liquid Cooled Systems 
Essentially an IT fan powered "Rear-Door Coil" system 


Each compute cabinet has a cooling coil extracting its 
generated heat from the air-stream before the next 
compute cabinet in row 


e Edison 30 compute cabinets (8/row) 
e Cori 64 compute cabinets (12/row) 


Å 


Air Flow 


| | 3j — EEE HN. i 


vn & Compute rá s 
Valve Cabinet idi 
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Controls to Save IT Fan Energy 


* Blower Cabinets (6 Fans per) 
* Variable from 2500 to 4000 rpm 
e Dynamic Speed option saves energy sensing on HPC load with the processor temp. 
* Cooling water controlled to exiting air temp. (Typ. 69 — 71°F) 
e Can move cooling load more onto liquid when water cooling conditions favorable 


Cori Blower Fan Speed 


RPM 


„1 ER. TT 


3.0K 


28K 
8/1 8/2 8/3 8/4 8/5 8/6 8/7 


= c0-0 Fan Speed) == c0-0 Fan Speed? = CDD Fan Speed3 == c0-0 Fan Speed4 == CDD Fan Speed5 == c0-0 Fan Speedé 


Cori Water Valve Position 


% Open 


ir 


s = 


10 25 
8/1 8/2 8/3 8/4 8/5 8/6 8/7 
== €0-0 Position e» ch O Command Power supply 
air loop (with P i 
front door closed) OWET SUPPLY: 
water coil 
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Record High Temperatures Test NERSC 


= Cooling towers alarmed 


M 
on low water 
temperature: 105F 


= Cray s worked but 


Record high requ ired careful 
wetbulb: 72.5F mon Ito ri ng 
* Compute job 


curtailments were 
initiated as a 
precaution 
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Record Temps- Watching the Crays 


Cori Water Temperature (Top 5 Cabinets) 


8/30 9/1 9/3 9/5 9/7 


= C0-0 Water In = c0-4 Water In == c11-0 Water In = c2-0 Water In 


Cori Water Valve Position 


9/9 
c3-0 Water In 


Supply water 
temperature reached 
78?F (above design 
target of 74°F) 


Cray SEDC provides 
extensive internal 
performance data 


Only some cabinets had 
to open valves all the 
way 

Almost no thermal 
throttling 


The Crays worked! 
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Cooling lowerWaterLevel Upgrade 


Accurate radar level sensor transmitters recently installed 


Mid July, 2018 
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The Future - Moving more towards 
Liquid 
= NERSC-9 Project Underway 
* 10MW power service 
u pgrade (to 22.5MW building capacity) 
* Will be entirely liquid cooled 
* 3 more cooling towers with 


plant infrastructure to be 
installed 


e Also looking into new 
alternatives like 
thermosyphon | 

e System installation forecast to EZ 
begin Summer 2019, fully up 
in 2020 


Iding: 


% 
19 ME cU 


Å — me ë e | 
Lessons & Conclusions 


* NERSC trend is even more towards liquid cooling 
* Hopper, (air cooled) and Edison (XC Series LC) retire in 2019 
* NERSC post 2019: Cori & NERSC 9 period of 2 systems, working towards a 
single liquid on-board cooled system 
* Cooling tower water use is very high and becoming a more important savings 
measure focus 


* WUE calculation project pending — Current estimate is 1.2 to 1.5 
liters/kWh (WUEsite) 


* Blowdown water saving measures are being implemented 
* Looking into State Point Liquid Cooling, new indirect evaporative system 


* Thermosyphon is under consideration (NERSC 9) 
= No space for Dry Air Coolers presently (Possibly NERSC 10) 


* High resolution instrumentation of liquid systems provide beneficial 
information for operation improvements, but not without work overhead 
* Data analysis can be difficult — Good visualization tools a necessity 
* Keeping the system running requires considerable admin attention 
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National Energy Research Scientific Computing Center 


Norm Bourassa, nibourassa OQlbl.gov 
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NERSC Optimization Project - Ongoing 


* Saved ^2,150,000 kWh & 150,000 gal water over the last year 
* Estimated 0.045 PUE improvement 
* 1,800,000 kWh Non-IT savings 
* 750,000 kWh IT savings 


Optimize Cooling Tower Fan and Pump . . 
Controls ` Still more savings on the 


table: "600,000 kWh/year 
— supply air fan 


optimization 
Reset Cooling Water Supply Temperature 400,000 400,000 —  Cray Dynamic Fan 
and Optimize CRAY Controls Speed Control 
Install Firmware to Enable ESS Mode for 350.000 e CW plant inter-tie 
UPSs F — Bypass valves for 
Replace Bypass Valves | L OOO > dl v 
7 |Cold Aisle Partial Containment | 10000|- | > S (ESA [ac 
reclaim 


Total 800,000 2,150,000 150,000 
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Liguid in the Rack: Liguid 
Cooling Your Data Center 


NREL ESIF Data Center 
Otto Van Geet, PE - NREL 


på ( H g Better Buildings” 
ner 


Otto Van Geet, PE 
Principal Engineer, NREL 
Otto.vangeet O nrel.gov 
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NREĽ s Dual Computing Mission 


= Provide HPC and related systems expertise to advance 
NREL's mission, and push the leading edge for data center 
sustainability 


= Demonstrate leadership in liquid cooling, waste heat 
capture, and re-use 


* Holistic "chips-to-bricks" approaches to data center 
efficiency 


* Showcase data center at NREL's Energy Systems 
Integration Facility (ESIF) 


= Critical Topics Include: 
* Liquid cooling and energy efficiency 


e Water efficiency 
Ae Sy Sl NO 
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Pla nning fora New Data Center 


Started planning for new data center in 2006 


* Based on HPC industry/technology trends, committed to direct liquid cooling 


* Holistic approach: integrate racks into the data center, data center into the facility, 


the facility into the NREL campus 


* Capture and use data center waste heat: office and lab space (now) and export to 


campus (future) 
= Incorporate high-power density racks— more than 60 kW per rack 


= Implement liquid cooling at the rack, no mechanical chillers 


* Usechilled beam for office/lab space heating. Low-grade waste heat use. 


* Considered two critical temperatures: 


e Information technology (IT) cooling supply —could produce 242C (759F) on 


hottest day of the year, ASHRAE "W2" class 


e IT return water—required 352C (959F) to heat the facility on the coldest day of 


the year 


Build the World's Most Energy Efficient Data Center 


A A 
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NREL Data Center 


Showcase Facility Data Center Features 

= ESIF 182,000 ft.2 research facility * Direct, component-level liquid cooling, 
= 10,000 ft.2 data center 249C (759F) cooling water supply 

= 10-MW at full buildout = 35-402C (95-1049F) return water (waste 


heat) is captured and used to heat offices 


LEED Platinum Facility, PUE < 1.06 and lab space 


NO mechanical cooling (eliminates 
expensive and inefficient chillers) 


* Pumps more efficient than fans 


* High-voltage, 480-VAC power distribution 
directly to high power density 60- to 80- 
kW compute racks 


Compared to a Typical Data Center 
= Lower CapEx— costs less to build 


= Lower OpEx— efficiencies save 


Integrated "Chips-to-Bricks" 


Utilize the bytes and the BTUSs! Approach 
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Future Campus Heating Loop 


Metncs 


"IT energy” — “Reuse energy” 
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700 forecast --> 


On-site 


"Annual Site Water Usage" 
WUE — — 5335 55 —5—d—— 


From 
electricity 
generation 


“IT energy" 


Water consumption (billion liters/y) 


the units of WUE are liters/ kWh 


0 
2006 2008 2010 2012 2014 2016 2018 2020 
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"Annual Site Water Usage” + "Annual Source Energy Water Usage” 


WUE source = “IT energy” 


“Annual Site Water Usage” 


WUEsource= ut enery" —— —Ó + [EWIF x PUE] 


where EWIF is energy water intensity factor 
ÆR. fy FN my SAME 
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Air-C ooled to Liquid -Cooled Racks 


Traditional air-cooled Require liquid-cooled when rack 
allow for rack power power densities in 5-80kW range, 
densities of 1kW-5kW have several options 
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Liquid-Cooled Server Options 


Cold Plates 
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Fanless Liquid -Cooled Server O ptions 


Fixed Cold Plate 


Wi 
4 
4 
A 
A 
A 
d 
A 
A 
A 


Direct Immersion 
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Data Center Water Distnbution 


——r To/ From 4th Floor 
Mechanical Room 
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Liquid Cooling —Consideratilons 


= Liquid cooling essential at high-power density 
= Compatible metals and water chemistry is crucial 


* Cooling distribution units (CDUs) 
e Efficient heat exchangers to separate facility and server liquids 
e Flow control to manage heat return 
e System filtration (with bypass) to ensure quality 


= Redundancy in hydronic system (pumps, heat exchangers) 


= Plan for hierarchy of systems 
* Cooling in series rather than parallel 
e Most sensitive systems get coolest liquid 


= At least 9596 of rack heat load captured directly to liquid 


C — € Better Buildings’ 
o Exchange SUMMIT 


Wl M 
Basic Hybrid System Concept 
KG Heat 4 DEE 


A A 
Wet Loop 


> or > 
Dry Sensible Cooler Sized for Design Day 
Thermal Duty 


95°F (35.0°C) 


Dry Loop 
Sized for Water Savings 


85°F (29.4°C) 75°F (23.9°C) 


95°F (35.0°C) 
Heat In 


=> 


Process Loop 


ed Wet when it’s hot, “dry” when it’s not. 


(Å N Condenser Water Pump 
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Improved WUE— Themnosyphon 


Water Vapor Q 


Q 


out 


4th Floor Mechanical 


Cooling Towers 
E. JAN Mechanical Space Cooling Tower 
(Indoors) HEX 


Monthly mean PUE for the ESIF Data Center 
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Node Racks 
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Power Usage Effectiveness 


2nd Floor Mechanical 


1.000 


Variable " 9 oo 8 8 nuu 
[odol] 4—— we um Pp oe oe d de SS > Ss a A A AS d Aë d d e 


Heat Energy ===> 


g 

3 

Š 
AAA 


3 Energy Recovery 
o HEX 


out 


—> Water => Air => Refrigerant 


ERW Loop y Sa di 
(Energy Recovery Water) m )) ( ni la 
4 e IN R E L Johnson H National 


Å. Controls Laboratories 
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ESIF Data Center Efficiency Dashboard 


e: NRE L ESIF HIGH PERFORMANCE COMPUTING DATA CENTER OUTDOOR | ^5 "Peer 785 Ff 


As of Tue Aug 7 10:27:29 MDT 2018 Provided values in °F and GPM Relative Humidity 442 o 
Lights & Plugs — Cooling Pumps HVAC — ITEquipmet — Reuse IT Equipment 
iiem - ( 1033 + 1827 + 615 + 583 + 93114 )/ 931.14 —— tas 
34 3 26 Fan Ponsa 
(kW) 
ERE " = = 
O -( 1033 + 1827 + 615 + 583 + 93114 - 66.60 )/ 931.14 == CT 
—— dano X — ee — áá——ÓÀ— NER — a ee EE 8 


Liquid Cooling 
Node CDU (Cooling 
Racks Distribution Unit) 


Air Cooling 


Dry Balb lemp 
766 


Heat 1ramsťer (kW) 


-271.2 121.5 


Energy Recovery Water (ERW) Loop 


Fan Walls ` Sec South Nor Where is Data bus Energy Going? —> Water 
Flow All Input Water 64.6 640 Airin Temp. 78.5 187 ESIF Building Heat 656 kW hk % 
DI? See Power Outdoors via Cooling Towers — 1215 kW 
m Out Coil Flow 33.6 400 Fan (QW) 21 21 via = E i - 
Heat Transfer (kW) Heat Iransfer (kW) Coil Pump 01 02 Heat Energy 389 417 Campus Building 55. KW —— e gerant 
558 -748 656 -660 Power (kW) Captured (AV) 


https://hpc.nrel.sov/cool/ 


Data Center Metnc s 


First year of TSC operation (9/1/2016-8/31/2017) 


Hourly average IT Load WUE = 0.7 liters/kWh 
= 666 kW (with only cooling towers, WUE — 1.42 liters/kWh) 
PUE — 1.034 ES Site Water Usage and Estimated Water Savings 
ERE = 0. 929 300,000 A A ; 1,135,624 
250,000 . m 4 WN 946,353 
v d Ň e Ň Ň : Ň Få 
Annual Heat Rejection eem | | : d | d d d ef: 
(© 150,000 E Ň 567,812 35 
Building Heat ` 100,000 | : d d d : : 378,541 
50,000 l - 8 i l | 189,271 
' LRS DSD DD E SSD ' 
d Y a O 
E Site Water Usage 7. Estimated Reuse Energy Water Savings 


N Estimated Thermosyphon Water Savings 


Thermosyphon WUE source = 5.4 liters/kWh 
VE source = 4.9 liters/kWh if energy from 


720 kW PV (10.592) is included 
using EWIF 4.542 liters/ kWh for Colorado 
AENEA "UD CE 


ia . 
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Notice 


This research was performed using computational resources sponsored by the U.S. 
Department of Energy's (DOE) Office of Energy Efficiency and Renewable Energy 
and located at the National Renewable Energy Laboratory under Contract No. DE- 
AC36-08GO28308. Funding provided by the Federal Energy Management 
Program. The views expressed in the presentation do not necessarily represent 
the views of the DOE or the U.S. Government. The U.S. Government retains and 
the publisher, by accepting the presentation for publication, acknowledges that 
the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide 
license to publish or reproduce the published form of this work, or allow others to 
do so, for U.S. Government purposes. 
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